Site Directed Management of Audio Components of Uploaded Video Files

ABSTRACT

A system, method and various software tools enable a video hosting website to automatically identified unlicensed audio content in video files uploaded by users, and initiate a process by which the user can replace the unlicensed content with licensed audio content. An audio replacement tool is provided that enables the user to permanently mute the original, unlicensed audio content of a video file, or select a licensed audio file from a collection of licensed audio, and insert the selected in place of the original audio.

CROSS REFERENCE TO RELATED APPLICATION

This application incorporates by reference U.S. Provisional ApplicationSer. No. 60/856,501 filed Nov. 2, 2006 in its entirety. This applicationis also related to U.S. patent application Ser. No. ______ filed on Feb.13, 2007 and entitled “Blocking of Unlicensed Audio Content in VideoFiles on a Video Hosting Website.”

TECHNICAL FIELD

The present invention relates generally to web-based video display andspecifically to software tools and methods for managing audio and othercontent in an online content distribution system, in relationship to thecontent owner's copyrights.

BACKGROUND

The sharing of video content on websites has developed in a worldwidephenomenon, supported by dozens of websites. On average, over 100,000videos are posted every day to various video hosting websites, and thisnumber is increasing, as the tools and opportunities for capturing videobecome easy to use and more widespread. In some cases, the individualswho post videos are unaware that others may have rights in the videocontent, audio content, or both or a particular video. For example, itis relatively simple to find copies of music videos by well knownartists, as well as homemade video that include audio soundtracks fromcopyrighted music works.

When a video is posted that contains audio content that is owned by acopyright holder such as music label or music publisher, or individual,that content owner has the right under current copyright law,particularly the Digital Millennium Copyright Act, to issue a “takedown”notice to the hosting website, requesting that it remove the identifiedvideo. Currently in the United States, the website must then remove theidentified video and provide an opportunity for the user who posted thevideo to challenge the notice. The handling of takedown notices resultsin increased overhead in terms of mechanisms to handle the notices anduser responses in order to ensure legal compliance. In addition, theremoval of posted videos is disruptive of the overall user experience onthe website.

SUMMARY OF INVENTION

Various systems, methods and software tools enable users who post videoson a video hosting website to manage the audio content of the videos,including replacing (“swapping”) audio content of a video, permanentlymuting the audio content, or entirely removing the video file from thesite. An audio replacement software tool is interoperative with thevideo hosting website to enable the user to replace unlicensed audiocontent, which would otherwise be subject to a takedown notice, forexample, with either licensed audio content or license-free content. Thevideo hosting website can thereby proactively avoid takedown noticesfrom being issued, and thereby reduce the costs and complicationsassociated with the compliance with the takedown notices. In addition,by enabling users to swap unlicensed content for licensed content, thesite can establish revenue sharing arrangements with the content ownersof the licensed content. The content owners benefit from increaseddistribution of their audio content, and in some cases receiveappropriate compensation for its use.

Another aspect of the system is the integration of the audio replacementtool and system support with a rights claiming management system. Therights claiming management system includes a content identificationsystem, which is adapted to process uploaded video content and identifywhether the audio content thereof is owned by a third party, such as amusic label or publisher. When the audio content is identified asbelonging to a third party that has not licensed its use on the videohosting site, the user who uploaded the content is notified of apossible copyright infringement, and directed to use an audio editingtool provided by the video hosting website to manage the audio contentof the file. The content owner may also be notified of the presence ofthe unlicensed content at this time or subsequently; a content claimrecord is created in a content claim database. This record can be usedto track claims for content ownership and revenue sharing.

The video file with the unlicensed content can be managed at the videohosting site in a number of different ways. First, the video can beposted for general access, but with a file setting (e.g., a “mute flag”)that indicates to a video player to mute the audio content duringplayback; in addition a message is displayed on the playback page (or inthe player) indicating that the audio content is being blocked. Thisallows other users to know why the audio is not being played, instead ofassuming that there is a problem with the player or their computer.Alternatively, the video content can be withheld from general accessuntil the posting user corrects the problem with the unlicensed audiocontent.

As noted above, the audio replacement tool provides the user with thecapability to replace the unlicensed audio content with licensedcontent, to remove the video from the site, or to permanently mute theaudio content during playback. Where the user uses the replacement toolto select and replace the unlicensed content with licensed content, theexisting unlicensed audio file is removed from the video file, and theselected licensed audio content is mapped to the video file. The claimrecord that indicated the presence of the unlicensed content, can beclosed, and a new claim record instantiated to indicate that an instanceof licensed content, along with an indicator of a revenue sharing policythat is associated with the audio content (alternatively, the existingclaim record can simply be updated to show the current license status).Once the licensed content is associated with the video file, the muteflag can be removed, and licensed audio content played back with thevideo when needed; the associated blocking message is also removed.

The audio replacement tool may be configured by the video hosting siteto be closed so that it can only access a restricted library of licensedcontent, or can be configured to be open, so that it can access anycontent available to the user via his computer (e.g., locally storedaudio files). The former configuration is useful to ensure that onlylicensed content is used to replace unlicensed content, so that thecontent providers benefit from guaranteed use of their content (andpotential revenue share). The latter configuration is useful to affordmaximum flexibility and artist freedom to the posting users. The audioreplacement tool can further be variably configured (e.g., open for someusers, closed for others) depending on other criteria (e.g., payments,user status, promotions, etc.).

The audio replacement tool can be provided in a number of differentways. One embodiment provides the audio replacement tool as abrowser-based tool (application) hosted by the video hosting website.Another embodiment provides the audio replacement tool as a standaloneapplication executing on the user's computer, and preferablypre-configured to access the libraries of licensed content on the videohosting site.

In some of the described embodiments, the audio replacement tool is usedin conjunction with the rights management system and the contentidentification system, following identification of unlicensed audiocontent therein when the user posts a video file. In another embodiment,the audio replacement tool can be used independently of the contentidentification system. Here, the user first uploads a video file (whichmay have either licensed audio content, unlicensed audio content, usercreated audio content, or no audio content) and accesses the audioreplacement tool to add (or replace) audio content to the video file.The user can select an audio file from a library of licensed audiocontent. The selected audio file is then combined with the video file toproduce a final video file. The final file is then posted for generalaccess. The content owner of the selected audio file is provided withthe appropriate revenue share for the use of the licensed content.

The features and advantages described in this summary and the followingdetailed description are not all-inclusive. Many additional features andadvantages will be apparent to one of ordinary skill in the art in viewof the drawings, specification, and claims hereof.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system architecture for providing audioreplacement for uploaded video content.

FIG. 2 is a process flow for an automatic takedown process

FIG. 3 is an illustration of the audio replacement tool.

FIG. 4 is an illustration of a web page showing the messaging associatedprovided with the audio content of a video file is blocked.

The figures depict various embodiments of the present invention forpurposes of illustration only. One skilled in the art will readilyrecognize from the following discussion that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles of the invention described herein.

DESCRIPTION OF EMBODIMENTS

It is to be understood that the Figures and descriptions of the presentinvention have been simplified to illustrate elements that are relevantfor a clear understanding of the present invention, while eliminating,for the purpose of clarity, many other elements found in typicalcommunication system and method of using the same. Those of ordinaryskill in the art may recognize that other elements and/or steps aredesirable and/or required in implementing the present invention.However, because such elements and steps are well known in the art, andbecause they do not facilitate a better understanding of the presentinvention, a discussion of such elements and steps is not providedherein. The disclosure herein is directed to all such variations andmodifications to such elements and methods known to those skilled in theart.

FIG. 1 is a block diagram of a system architecture in accordance withone embodiment of the present invention. As shown in FIG. 1, a videohosting website 100 includes a front end server 140, a rights managementsystem 130, an audio replacement tool 120, a video server 110, an ingestserver 115, and various databases, including a user database 150, alicensed content database 200, a video database 190. Many conventionalfeatures, such firewalls, load balancers, application servers, failoverservers, site management tools, and so forth are not shown so as notobscure the features of the system. A suitable website 100 forimplementation of the system is the YOUTUBE™ website, found atwww.youtube.com; other video hosting site are known as well, and can beadapted to operate according the teaching disclosed herein. It will beunderstood that the term “web site” represents any method of uploadingand downloading content and is not intended to be limited to contentuploaded or downloaded via the Internet or the HTTP protocol. Thevarious servers can be implemented as a single piece of software orhardware or as multiple pieces of software or hardware. In general,functions described in one embodiment as being performed on the serverside can also be performed on the client side in other embodiments ifappropriate. In addition, the functionality attributed to a particularcomponent can be performed by different or multiple components operatingtogether.

A client 170 executes a browser 171, and can connect to the front endserver 140 via a network 180, which is typically the Internet, but mayalso be any network, including but not limited to a LAN, a MAN, a WAN, amobile, wired or wireless network, a private network, or a virtualprivate network. While only a single client 170 and browser 171 areshown, it is understood that very large numbers (e.g., millions) ofclients are supported and can be in communication with the website 100at any time. The browser 171 include a video player (e.g., Flash™ fromAdobe Systems, Inc.), or any other player adapted for the video fileformats used in the site 100. A user can access a video from the site100 by browsing a catalog of videos, conducting searches on keywords,reviewing playlists from other users or the system administrator (e.g.,collections of videos forming channels), or viewing videos associatedwith particular user group (e.g., communities). A browser 171 can alsoaccess a video file indirectly, via an embedded video 177 that isaccessed via an embedded hyperlink in a third party website 175.

Users of the clients 170 and browser 171 can upload content (which caninclude, for example, video, audio, or a combination of video and audio)to site 100 via network 180. The uploaded content is processed by aningest server 115, which processes the video for storage in the videodatabase 190. This processing can include format conversion,compression, metadata tagging, and other data processing. An uploadedcontent file is associated with the uploading user, and so the user'saccount record is updated in the user database 150 as needed.

For purposes of convenience and the description of one embodiment, theuploaded content will be referred to a “videos” or “video files”, but nolimitation on the types of content that can be uploaded are intended bythis terminology. Each uploaded video is assigned a video identifier(id) when it is processed by the ingest server 115. The ingest server115 communicates with the rights management system 130 to determinewhether the uploaded video is subject to rights of content owners otherthan the user who uploaded the video.

The video database 190 is used to store the ingested videos. Thelicensed content database 200 stores encoded audio files and associatedmetadata, provided by their respective content owners. The audio filesare can be encoded at .mp3 files at 64 kbps, mono, 22.1 KHz, or betterquality (e.g., 128 kbps, stereo, 44.2 KHz). The metadata for each audiofiles includes an ISRC (or custom identifier), artist, song title,album, label, genre, time length, and optionally geo-restrictions thatcan be used for data collection or content blocking on a geographicbasis.

The rights management system is one means for processing the video fileto determine whether the audio content of the video file is anunauthorized use of a content owner's licensed content. In oneembodiment, the rights management system 130 uses two types ofidentification processes to make this determination. First, the rightsmanagement system 130 generates a hash (e.g., MD5, SHA-n, etc.) of thevideo file itself, to produce a file signature. This signature (or hashcode) is compared with an existing collection of file signature of knownlicensed content. In this embodiment, the licensed content would beaudio files from music labels, publishes, independent artists, or thelike. The file signature can correspond to any licensed content,including content that is available on the site 100 as well as contentthat is not available on the site, but only through other channels. Thisfeature allows the system 130 to identify unlicensed audio content,regardless of its availability within the site 100. If the filesignature matches the signature of a licensed content file, then is itis determined that audio content is unlicensed, and hence aninfringement of the content owner's rights in the licensed content file.In this case, the content owner is determined for the licensed content,and the owner's policy for revenue sharing or block is retrieved fromthe claim database 134. If the policy is to block the content, then thevideo file is marked as restricted, and thus not enabled for generalaccess in the video database 190; alternatively the file is not loadedinto the database 190 and is discarded. If the policy indicates revenuesharing, then a claim record is established in the claim database 134,indicating the particular video ID content, content owner, and policy. Asubsequent accounting process is run periodically to allocate revenue tothe content owner for the use of the audio content. The parameters forallocation of revenue are stored in association with the content owner'spolicy record in the licensed content database 200.

If there is no match for the file signature, and the second level ofidentification is provided by the content identification system 131. Thecontent identification system 131 employs an audio fingerprint on theaudio portion of the video file. The audio fingerprint is based onanalysis of spectral content of the audio file (e.g., frequency,amplitude, phase, etc.). A suitable content identification system forprocessing audio fingerprints is available from Audible MagicCorporation, of Los Gatos, Calif. A fingerprint database 132 ismaintained that contains the fingerprints of licensed content from knowncontent providers. The fingerprint database 132 can contain fingerprintsprovided by directly by the content owners, generated by the provider ofthe content identification, or generated internally by the video hostingwebsite 100 using the appropriate analysis tools. The fingerprints inthe database 132 can correspond to any licensed content, includingcontent that is available on the site 100 as well as content that is notavailable on the site, but only through other channels. This featureallows the content identification system 131 to identify unlicensedaudio content, regardless of its availability within the site 100. Amore detailed description of the file signature and fingerprint matchingprocess is described in the above referenced related application.

If the fingerprint of the audio content of the video file does not matcha known fingerprint, then is determined that the audio content is not anunauthorized copy of a licensed content. The video file is then enabledin the video database 190 for general access by browsers. The videoserver 110 can access the stored file and provide it to a requestingbrowser. If the fingerprint of the audio content of the video file doesmatch a known fingerprint, then the automatic audio replacement processis initiated.

FIG. 2 illustrates one embodiment of the automatic audio replacementprocess, which can be executed by the rights management system 130, orboth other software components, as desired by the system administrator.For the purposes of this explanation, FIG. 2 begins in response to thepositive identification of either a file signature or the audiofingerprint. Having identified the licensed content from the filesignature or fingerprint, the rights management system 130 hasinformation that identifies the owner of the content, and so retrieves205 the owner's policy information for handling instances of unlicensedcontent. Generally, the policy will indicate whether to block the audiofile from playback entirely, or allow a revenue share for the audiocontent. Optionally, the content owner can also be notified 207, forexample, by an email; alternatively, the content owner can be notifiedin a batch process, either periodically (e.g., once daily, weekly,monthly), or after a preset number of matching fingerprints (e.g., each100 matches), or the content owner can be given access to the claimdatabase 134 by which they can inspect and determine which content hasbeen identified for claims.

If the policy for the content owner is indicated to be revenue sharing,then the video file and its audio is allowed to be posted and accessedwithout restriction. A claim record is established in the claim database134, indicating the particular audio content, content owner, and policy.

Where the content owner's policy is to block the audio content, then adifferent set of operations is engaged to create an automatic takedownof the identified audio content. An email is automatically generated andsent 204 to the uploading user, identifying the uploaded file, theidentified audio content (e.g., song name, artist, date, publisher,etc.) and informing the user that the uploaded content has been found tomatch a known work of licensed content, and that it infringes thecontent owner's copyright in such content. The email instructs the userto login to the site 100 and correct the problem through variousalternative options. The user is given a period of time (e.g., 15 daysor even 0 days) by which to take action to correct the problem. (Thenumber of days for action can be varied, depending on considerationssuch as follow of identification, user feedback, or the like). The emailalso includes links to various copyright information pages 194 (FIG. 1),which further explains the issues pertaining to copyright law and theunauthorized use and distribution of audio content. The user accessesthis email through an email client 206. The rights management system 130also sets a flag in the user's account that there is an outstandingblock on an uploaded video file. If the does not undertake one of theediting action (mute audio, replace audio, remove file) within apredetermined amount of time (e.g., 10 days), then video file isretrieved from the database 190, the audio content is automaticallyremoved from the video file, and the modified video file is stored backto the database 190. The mute flag can then be removed since the videowill now play without any audio at all; this approach is beneficial toprevent a malicious user from removing the mute flag or modifying avideo play to ignore it. Alternatively, instead of providing the user alimited amount of time to act, the audio file is immediately removed,and the user is notified and offered the option to replace the audio asdescribed below.

In addition, the video file is updated in the video database 190 withthe mute flag (or other suitable signal) set to indicate to a compatiblevideo player to mute the audio upon playback. When a user access thevideo, the page for that video includes a message that indicates thatthe audio is muted due to it being an unauthorized use of the content.FIG. 4 illustrates an example page 400 with such a message 402. Themessage can also appear in the video playback area 404 as video, text orgraphics, in conjunction with (e.g., prior to or during) the playback ofthe video. Optionally, an audio notification file with a recordedmessage indicating that audio content of the video has been muted, canbe played during (or before) the video file is played; this feature isparticularly beneficial for video files that are embedded in third partypages. When a user (any user) makes a request for the video file, thevideo server 110 begins streaming the video file to the user's browser171 (which executes a video player). The video player detects the muteflag, and thus mutes the audio while playing back the video file.Another alternative approach is to process the audio content eitherbefore or during playback to render it no longer recognizable, forexample by scrambling, garbling, or otherwise distorting the audiocontent; this can be done either through manipulate of the encoded audiodata packets, or through processing of the audio content in thefrequency domain. An appropriate message 402 can likewise be provided inthis instance to alert the viewer. Another alternative approach is toreplace the unlicensed audio with a computer generated melody thatmatches a musical characteristic of the audio content, such as its tempo(but preferably does not use samples, voices, or pitches). Thisreplacement is done generally after the unlicensed audio content hasbeen detected, and prior to the video being edited by the uploadinguser. This provides the viewing user with a limited audio experiencewithout resulting in an impermissible use of the licensed audio content.

As indicated above, in some instances a video file is embedded in a pageon a third party website, and plays in that page, via a hyperlink to thefile. In these cases, since the video is playing in the third partypage, the message that is otherwise present on the host site 100indicating that the audio is being blocked and would not be available.Accordingly, for these embedded video files, in one embodiment, theentire video is blocked from playback; alternatively a video filecontaining a message can be played first indicating that the audioportion of the file is being muted, or with the audio notificationmentioned above.

The user who uploaded the video file subsequently logins 208 to thevideo hosting website 100. Upon login the site 100 checks the user'saccount, and determines that there is outstanding block on the upload.The site thus loads for the user the copyright infringement notice page220 which describes to the options for resolving the blocked audiocontent. The user can dispute 222 the claim that the content isunauthorized (i.e., infringes the content owner's copyright in theaudio), in which case the user is taken through a series of pages 224,226 describing the dispute process in which the user can file acounter-notification under the DMCA. The user is required to confirmthat he has read and accepted the information on each page, in order toreduce the number of instances of false disputes.

If the user decided to proceed to a dispute, the user is presented withan online form in which he must input the following information:

-   -   a physical or electronic signature of the subscriber (e.g.,        typing the user's full name into a signature box);    -   identification of the audio content that has been removed or to        which access has been disabled and the location at which the        content appeared before it was removed or access to it was        disabled; this is the video ID of the video;    -   a statement under penalty of perjury that the user has a good        faith belief that the audio content was removed or disabled as a        result of mistake or misidentification of the material to be        removed or disabled; here is checkbox is provided for the user        to check indicating acknowledgement of the statement;    -   The user's name, address, and telephone number, and a statement        that the user consents to the jurisdiction of Federal District        Court for the judicial district in which the address is located,        or if the user's address is outside of the United States, for        any judicial district in which the service provider may be        found, and that the user will accept service of process from the        person who provided notification or an agent of such person;        here the user enters their personal information.

Once the user submits the above information in the counter-notificationform, the site 100 checks that the video ID input by the usercorresponds to a video owned by the user. The site 100 also checks thatthe video ID has not been filed in previous counter-notification,thereby avoiding duplicate claims. The site 100 further checks thatcontent owner of the audio content for the video ID has a policy whichblocks the use of the content. If all of these checks are passed, then aconfirmation page 228 is presented to the user to confirm that thecounter-notification has been made. If any of these last checks fail,the user is provided with an appropriate error message.

Once a counter-notification is filed, the site 100 marks the user'svideo to withhold it from general access and playback for predeterminedperiod of time (e.g., 15 days). An email is then sent 230 to the contentowner which includes the information in the counter-notification (and/orthe completed counter-notification form itself), so that the contentowner is informed that the user is disputing the content owner's claimof infringement; the content owner is given the predetermined period oftime in which in which take further action under the DMCA, and sendnotification of such action to the site 100. If the content owner failsto act with the prescribed time period, then the counter-notification isconsidered valid, the video is released from its hold, and can beaccessed without restriction. The site 100 sends an email to the user inthis instance advising him that the video has been released. If thecontent owner takes legal action, then the user is likewise notified byan email from the site 100. In this circumstance, the site 100 removesthe video from the system to prevent its further playback.

In most cases, the user will not proceed to a counter-notification, andinstead will act to edit the video. The user has several differentediting actions that they can apply to the video file, includingremoving 232 the video file so that it is no longer available, muting236 the audio content, or editing 234 the audio content to select alicensed audio content for the file. Should the user decide to remove232 the audio content, a confirmation page 238 is provided to the user;after confirmation, the video is removed from the video database 190.The user is directed 240 to their “My Videos” page, which contains alisting of their posted video. A message is presented on this pageindicating that the posted video has been removed.

Should the user decide to mute 236 the audio content, anotherconfirmation page 242 is presented to confirm the user's decision. Thewebsite 100 invokes a utility tool which then removes the unlicensedaudio content from the video file, which has the same effect aspermanently muting the audio. This action is irreversible.Alternatively, the unlicensed audio content can be left in the videofile, with the audio mute flag set to mute the audio playback. The useris directed 244 to their My Videos page, and a message is provided therestated that the audio content for the video has been permanentlyblocked. In one embodiment, the user can later use the audio replacementtool 120 to insert licensed audio content into the video file.

Should the user decide to edit 234 the audio content, the audioreplacement tool 120 is loaded (for example, as part of the Javascriptof a page) or invoked (as standalone application or browser plugin)depending on the implementation. The audio replacement tool 120 is onemeans for enabling the user to replace the audio content of a video withaudio content from other sources including a library of licensed audiocontent. Referring to FIG. 3 there is shown one embodiment of an audioreplacement tool 120 as it would appear to the user. The audioreplacement tool 120 includes a video display window 302, which is onemeans by which the user can view the video (below the video display area302 are playback control, a timer, a volume control, and size controls).The user provided title 302 of the video is indicated above the videodisplay window 302. Also listed below the video display window 302 isthe name 310 of the identified, unlicensed audio content, its length312, and the length 314 of the user's video. To the right thereof is acontrol 316 for permanently muting the audio, should the user decide atthis time to mute the audio; selection of this option is one means thatoperates to remove the unlicensed audio content from the video file, asdescribed above. On the right side of the tool 120 there is shown alisting 304 of available licensed audio content, and is one example of ameans by which the user can select a file to replace the unlicensedaudio content; the listing 304 will generally include licensed contentthat is available from the licensed content database 200. The listingmay also include a list of freely available audio, such as audio in thepublic domain or available through a Creative Commons license or otheropen or free license; this option is beneficial to artists who areseeking further exposure to their music. The listing 304 includes thename 318 of the artist, the title 320 of the audio content, and itslength 322. Also provided is audio preview control 306 that allows theuser to playback a portion (or all) of the selected file; this featureallows the user to hear the selected content and decide if it isappropriate for their video. A control button 308, “Use Song” isprovided by which the user selects which audio content to use forreplacement.

The illustrated embodiment of the audio replacement tool 120 in FIG. 3is only one example of the audio replacement tool 120. In anotherembodiment, the content listing 304 includes a search tool, which is onemeans for searching for audio content by artist name, song name, albumname, year, or genre. In addition, the listing 304 can be made part of acategory browser, which is one means that allows the user to browse alibrary of audio content by any of the foregoing attributes.

When the user selects a particular licensed audio file for replacement,that audio content is mapped to the video file for playback. It shouldbe noted that the user is only allowed to select audio for playback with(and only with) the video file as the video is played back in astreaming manner; the user cannot select licensed audio content forplayback without any video, nor can the user enable the audio content tobe independently downloaded by other users at a later time. The user canpreview the edited video file, with the selected audio, to determine ifit acceptable, via the preview button 324. This causes the video portionof the user's video to be played back in conjunction with the selectedaudio content; the original audio content that was part of the videofile is muted.

If the selected audio content is acceptable to the user, the user thenpublishes the completed video file via the publish button 326. Theoriginal audio content of the file is then stripped, and the selectedaudio is stored into the video file, as required for the file format. Inone embodiment, the file is “flattened” into a single Flash FLV file inwhich the video and audio are merged. Depending on the configuration ofthe audio replacement tool 120, this can be done either internally byappropriate video and audio encoder in the audio replacement tool 120,or by providing to the video hosting site 100 an identifier of theselected audio content, which is then processed in conjunction with thevideo file by the appropriate video and audio encoder at the site 100;either mechanism provides a suitable means for replacing the unlicensedaudio content with selected audio content. The updated video file isthen stored back to the video database 190. Playback of the edited videofile appears like other video content in the database 190.

The audio replacement tool 120 can be configured to provide variouslevels of user control as to the mapping. The audio replacement tool 120can be configured to simply map the audio content to the video file fromthe beginning thereof, so that both the audio and video portions havethe same starting point. If the audio file is shorter in duration thenthe video file (e.g., the audio is 2:00 minutes and the video is 3:00minutes), then when the audio file is ended, the remainder of the videois silent. If the audio file is longer than the video file, then theaudio file is truncated to the same length of as the video file.Alternatively the audio replacement tool 120 can be configured to offsetthe start the audio file to any point in the video file. In oneembodiment, no farther audio editing tools are provided to the user.This restriction is beneficial to the content owners as it ensures theintegrity of the audio file.

In another embodiment, one or more various editing tools and effects canbe provided to the user in the audio replacement tool 120. First, theaudio replacement tool 120 can include a loop setting, by which theaudio file can be set to loop for the duration of the video file. Otheraudio editing tools or effects can be provided as well, such asequalization controls, volume control, pitch change, reverb, limiters,and the like.

Another aspect of this embodiment is to only allow a limited,predetermined number licensed audio files to be inserted into a videofile, regardless of the length of the video file. In one embodiment, theaudio replacement tool 120 is restricted to allow only a single audiofile to be inserted. This feature is beneficial to the content owners soas to ease the administration of the revenue sharing payments that comefrom the playback of the video. Accordingly, in an embodiment allowingonly a single audio file, the audio replacement tool 120 is configuredso that if the user first selects one audio file from the listing 304,and then selects a second file, the first file is removed (i.e., notused) and the second file is used instead. In addition, the audioreplacement tool 120 is configured so that the user cannot save ordownload a licensed audio content to their local client 170. Thisprevents the user from obtaining an unauthorized copy of the audiocontent. The audio replacement tool 120 may be configured by the videohosting site 100 to be closed so that it can only access the audiocontent in the licensed content 200, which then appears in listing 304.Alternatively, the audio replacement tool 120 can be configured to beopen, so that it can access any content available to the user via hiscomputer (e.g., locally stored audio files). In this embodiment, theaudio replacement tool 120 is configured to allow users to replace ormix the licensed audio in the video with the audio of any other videosthat any other user has uploaded to the site. The audio replacement tool120 can be further configured to allow users to dynamically choosebetween multiple audio tracks when playing back the same video, forexample French or Spanish versions of the same movie clip, or playingback a politician's speech with a comedic audio track. Finally, theaudio replacement tool 120 can further be configured to allow the userto record audio “comments” or karaoke by means of a microphone attachedto their client device and appropriate sound card.

Referring again to FIG. 2, after the user has removed the video file ormuted the audio the rights management system 130 updates 250 the claimsdatabase 134. This can be done by removing the previous claim record, orupdating the claim record to indicate the user's action, and therebyprovide an audit trail. Where the user has selected to use licensedcontent, a new claim record is instantiated, indicating the contentowner, the licensed content and the revenue share policy flag.

Once the user has included the licensed audio content in the video filewith the audio replacement tool 120, they are directed back to a videoplayback page 246 for watching the video file. This page is updated toinclude attribution information for the artist of the audio content,including the artist's name, album, song name, and song information.Additionally, hyperlinks are provided on the page to a music commercesite where the user can purchase the audio content for separatedownloads. This video playback page 246 and its information and linksare displayed to any user who accesses the video file, not just the userwho uploaded it.

The above described process of detecting unlicensed audio content isinvoked when a user uploads a video file. A similar process ofidentification of unlicensed audio can be engaged for videos that arealready uploaded into the video database 190. Here, a selection ofvideos that are in the video database 190 are processed by the contentidentification system 131 for fingerprint matches to licensed audiocontent. Video files that having a matching fingerprint at flagged andthe process described for FIG. 2 is initiated. For these existing video,there may be legacy information, such as comments, view counts, ratings,tags, and other information associated with the video. This legacyinformation is maintained when the video is either muted or it has itsaudio content replaced.

The audio replacement tool 120 can be configured as well for use anytime a user uploads a video file and would like to add licensed audiocontent to it directly, instead of using whatever audio content is inthe video file, whether their own audio content or other licensedcontent. In this embodiment, after the user uploads a video file, he caninvoke the audio replacement tool 120, and then select a licensed audiocontent for inclusion in the video file. This selected audio file can beused to replace any existing audio for the video file, or mixed in withit as an overdub. For example, the user can replace an existing licensedaudio content with other licensed audio content. As before, a new claimrecord is then created in the claim database 134, so that the contentowner of the licensed audio content receives the appropriate revenueshare from its use. As a further extension of this embodiment, the audioreplacement tool 120 can be configured to be open, so that the user canalso select from audio tracks available on their computer or othernetwork locations. In this configuration, the selected audio file isprocessed by the content identification system 131 to determine if it islicensed content. If the audio file is determined to be licensedcontent, then the owner of the licensed content is identified from thefingerprint information, and the revenue sharing policy is determined.Where the policy indicates that the content owner allows for revenuesharing, then the audio content is included in the user's video. If thepolicy indicates that the content should be blocked, then the user isimmediately notified that they cannot use the selected content, andgiven the option to select other content.

Reference in the specification to “one embodiment” or to “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiments is included in at least oneembodiment of the invention. The appearances of the phrase “in oneembodiment” in various places in the specification are not necessarilyall referring to the same embodiment.

Some portions of the above are presented in terms of methods andsymbolic representations of operations on data bits within a computermemory. These descriptions and representations are the means used bythose skilled in the data processing arts to most effectively convey thesubstance of their work to others skilled in the art. A method is here,and generally, conceived to be a self-consistent sequence of steps(instructions) leading to a desired result. The steps are thoserequiring physical manipulations of physical quantities. Usually, thoughnot necessarily, these quantities take the form of electrical, magneticor optical signals capable of being stored, transferred, combined,compared and otherwise manipulated. It is convenient at times,principally for reasons of common usage, to refer to these signals asbits, values, elements, symbols, characters, terms, numbers, or thelike. Furthermore, it is also convenient at times, to refer to certainarrangements of steps requiring physical manipulations of physicalquantities as modules or code devices, without loss of generality.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“determining” or “displaying” or “determining” or the like, refer to theaction and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system memories orregisters or other such information storage, transmission or displaydevices.

Certain aspects of the present invention include process steps andinstructions described herein in the form of a method. It should benoted that the process steps and instructions of the present inventioncan be embodied in software, firmware or hardware, and when embodied insoftware, can be downloaded to reside on and be operated from differentplatforms used by a variety of operating systems.

The present invention also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, magnetic-opticaldisks, read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, application specific integratedcircuits (ASICs), or any type of media suitable for storing electronicinstructions, and each coupled to a computer system bus. Furthermore,the computers referred to in the specification may include a singleprocessor or may be architectures employing multiple processor designsfor increased computing capability.

The methods and displays presented herein are not inherently related toany particular computer or other apparatus. Various general-purposesystems may also be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear from the description below.In addition, the present invention is not described with reference toany particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof the present invention as described herein, and any references belowto specific languages are provided for disclosure of enablement and bestmode of the present invention.

While the invention has been particularly shown and described withreference to a preferred embodiment and several alternate embodiments,it will be understood by persons skilled in the relevant art thatvarious changes in form and details can be made therein withoutdeparting from the spirit and scope of the invention.

Finally, it should be noted that the language used in the specificationhas been principally selected for readability and instructionalpurposes, and may not have been selected to delineate or circumscribethe inventive subject matter. Accordingly, the disclosure of the presentinvention is intended to be illustrative, but not limiting, of the scopeof the invention.

1. A computer implemented method of identifying and replacing unlicensedaudio content in video files uploaded to video hosting website, themethod comprising: receiving a video file uploaded from a user at avideo hosting website, the video file including audio content;processing the audio content to determine whether the audio content isan unlicensed audio content that is owned by a known content owner otherthan the user; and responsive to the audio content being an unlicensedaudio content, receiving from the user an instruction to replace theaudio content with a licensed audio content selected by the user.
 2. Themethod of claim 1, further comprising: responsive to receiving from theuser an instruction to replace the audio content with a licensed audiocontent selected by the user, removing the unlicensed audio content fromthe video file, and inserting the selected licensed audio content. 3.The method of claim 2, further comprising: displaying a video playbackpage for the video file, the video file including the licensed audiocontent; displaying on the video playback page attribution informationfor the licensed audio content; and displaying on the video playbackpage a hyperlink to an e-commerce site from which the licensed audiocontent can be purchased.
 4. The method of claim 1, further comprising:providing to the user a list including licensed audio content; andenabling the user to select content at least one audio file from thelist of licensed content.
 5. The method of claim 4, wherein the listincludes freely available audio content, from which the user can selectaudio content to replace the unlicensed audio content.
 6. The method ofclaim 1, wherein receiving from the user an instruction to replace theaudio content with a licensed audio content selected by the user furthercomprises: executing an audio replacement tool that enables the user toselect the licensed audio content.
 7. The method of claim 6, whereinexecuting an audio replacement tool further comprises: executing theaudio replacement tool as a hosted application on the video hostingwebsite.
 8. The method of claim 6, wherein executing an audioreplacement tool comprises: executing the audio replacement tool as anapplication in client device of the user.
 9. The method of claim 6,wherein audio replacement tool is a closed audio tool with access onlyto audio content managed by the video hosting website.
 10. The method ofclaim 6, wherein the audio replacement tool is configured to allow theuser to select only a single audio content file to replace theunlicensed audio content.
 11. The method of claim 6, wherein the audioreplacement tool is configured to allow the user to select at least twoaudio content files to replace the unlicensed audio content, and to mixthe selected audio content files.
 12. The method of claim 6, wherein theaudio replacement tool is configured to allow the user to record audiocontent from a client device to replace the unlicensed audio content.13. The method of claim 1, further comprising: responsive to the audiocontent being an unlicensed audio content, notifying the user of theunlicensed audio content.
 14. The method of claim 1, further comprising:responsive to the audio content being an unlicensed audio content,setting the video file to instruct a video player to automatically mutethe audio content when the video file is played back.
 15. The method ofclaim 1, further comprising: responsive to the audio content being anunlicensed audio content, receiving from the user an instruction topermanently mute the audio content of the video file.
 16. The method ofclaim 15, further comprising: responsive to receiving from the user aninstruction to permanently mute the audio content of the video file,removing the audio content of the video file.
 17. The method of claim 1,further comprising: responsive to the audio content being an unlicensedaudio content, receiving from the user an instruction to remove thevideo file so as to prevent playback of the video.
 18. The method ofclaim 1, further comprising: receiving a plurality of video files beinguploaded by a user at the video hosting site.
 19. The method of claim 1,wherein processing the audio content comprises: generating a filesignature from the video file containing the audio content; and matchingthe file signature with the file signatures of licensed audio contentfrom known content owners.
 20. The method of claim 1, wherein processingthe audio content comprises: generating an audio fingerprint from thespectral characteristics of the video file containing the audio content;and matching the audio fingerprint with the audio fingerprints oflicensed audio content from known content owners.
 21. The method ofclaim 1, wherein processing the audio content further comprises:determining a revenue sharing policy of the content owner for theunlicensed audio content; and responsive to the revenue policy,determining whether to block the unlicensed audio content or allow theunlicensed audio content.
 22. A computer implemented method of editing avideo file in a video hosting website, the video uploaded to the site bya user, the method comprising: executing an audio replacement tool, theaudio replacement tool operating on the video file, the video fileincluding an original audio content portion; providing within the audioreplacement tool a listing of a plurality of licensed audio content;receiving a selection of one of the licensed audio content; replacingthe original audio content from the video with the selected licenseaudio content into the video file; and storing the video file with thelicensed audio content, so as to allow other users to access the videofile and licensed audio content for playback.
 23. A video hostingwebsite, comprising: an ingest server configured to receive from a userclient device an uploaded video file, the video file containing an audiocontent; a video database, coupled to receive from the ingest serveruploaded video files, and provide access to the uploaded video file inresponse to playback request; a rights management system coupled toingest server to receive an uploaded video file and configured todetermine whether the audio content of the video file is an unlicensedaudio content belonging to a content owner other than the user; alicensed audio content database; and an audio replacement tool coupledto the video database to receive a selected video file having anunlicensed audio content, provide a listing of licensed audio contentfrom the licensed audio content database, receive a selection of one ofthe licensed audio content, replace the audio content of the video filewith the selected license audio content, store the video file with thelicensed audio content in the video database, so as to allow other usersto access the video file and licensed audio content for playback.
 24. Anaudio replacement tool, adapted to interoperate with a video hostingwebsite, and comprising: a display area for displaying a video file, thevideo file including unlicensed audio content; an audio content listingarea for listing a plurality of audio content files available to replacethe unlicensed audio content, the listing of audio content received fromthe video hosting website; means for selecting one of the plurality ofaudio content files in the listing area; and a video and audio encoderadapted to receive the selected audio content file and replace theunlicensed audio content with selected audio content by encoding theselected audio content into the video file.